Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor/db #749

Merged
merged 73 commits into from
Jan 29, 2024
Merged

Refactor/db #749

merged 73 commits into from
Jan 29, 2024

Conversation

datenangebot
Copy link
Collaborator

@datenangebot datenangebot commented Dec 20, 2023 •
edited
Loading

closes #498

This should be tested carefully as it bring breaking changes with it.

  • Develop new db structure
  • Testing, testing, testing
  • Migration script for updating from old db schema

@juliusknorr juliusknorr self-requested a review December 21, 2023 12:39
juliusknorr
juliusknorr requested changes Dec 22, 2023
View reviewed changes
Copy link
Member

@juliusknorr juliusknorr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First round of feedback, but looks quite good already 👍

lib/Migration/Version000700Date20230916000000.php Show resolved Hide resolved
lib/Helper/ColumnsHelper.php Outdated Show resolved Hide resolved
lib/Helper/ColumnsHelper.php Outdated Show resolved Hide resolved
lib/Migration/Version000700Date20230916000000.php Outdated Show resolved Hide resolved
lib/Db/ColumnMapper.php Outdated Show resolved Hide resolved
lib/Db/Row2Mapper.php Show resolved Hide resolved
lib/Db/Row2Mapper.php Show resolved Hide resolved
lib/Db/Row2Mapper.php Outdated Show resolved Hide resolved
lib/Db/Row2Mapper.php Outdated Show resolved Hide resolved
lib/Migration/Version000700Date20230916000000.php Show resolved Hide resolved
@datenangebot
Copy link
Collaborator Author

Well, I think this is good for testing, testing, testing...

@juliusknorr juliusknorr added enhancement New feature or request 2. developing Work in progress labels Jan 8, 2024
@datenangebot
Copy link
Collaborator Author

datenangebot commented Jan 8, 2024 •
edited
Loading

Note to myself:

blizzz
blizzz reviewed Jan 8, 2024
View reviewed changes
Copy link
Member

@blizzz blizzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not see everything yet, but see some first comments. Generally, I am lacking a birds view, yet, need to get more familiar with the codebase.

lib/Db/ColumnMapper.php Outdated Show resolved Hide resolved
lib/Db/RowCellNumber.php Show resolved Hide resolved
lib/Db/RowCellNumberMapper.php Outdated Show resolved Hide resolved
lib/Helper/ColumnsHelper.php Outdated Show resolved Hide resolved
lib/Db/RowMapper.php Outdated Show resolved Hide resolved
lib/Db/RowMapper.php Outdated Show resolved Hide resolved
lib/Db/RowMapper.php Outdated Show resolved Hide resolved
lib/Db/RowMapper.php Outdated Show resolved Hide resolved
lib/Db/RowMapper.php Outdated Show resolved Hide resolved
blizzz
blizzz reviewed Jan 9, 2024
View reviewed changes
Copy link
Member

@blizzz blizzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second part of my review, left a couple of comments.

lib/Migration/NewDbStructureRepairStep.php Show resolved Hide resolved
lib/Migration/Version000700Date20230916000000.php Show resolved Hide resolved
lib/Service/RowService.php Outdated Show resolved Hide resolved
lib/Service/RowService.php Outdated Show resolved Hide resolved
lib/Service/RowService.php Outdated Show resolved Hide resolved
lib/Command/Clean.php Outdated Show resolved Hide resolved
lib/Command/TransferLegacyRows.php Outdated Show resolved Hide resolved
lib/Command/TransferLegacyRows.php Outdated Show resolved Hide resolved
lib/Command/TransferLegacyRows.php Outdated Show resolved Hide resolved
lib/Command/TransferLegacyRows.php Outdated Show resolved Hide resolved
@datenangebot datenangebot force-pushed the refactor/db branch 2 times, most recently from ba4de71 to 91feee4 Compare January 12, 2024 10:24
@datenangebot datenangebot marked this pull request as ready for review January 12, 2024 11:40
blizzz
blizzz reviewed Jan 15, 2024
View reviewed changes
Copy link
Member

@blizzz blizzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few comments

.github/workflows/app-upgrade-mysql.yml Show resolved Hide resolved
lib/Db/Row2.php Outdated Show resolved Hide resolved
lib/Service/RowService.php Outdated Show resolved Hide resolved
@blizzz
Copy link
Member

blizzz commented Jan 17, 2024 •
edited
Loading

I tried an upgrade with a handful of tables. The upgrade itself seemed to be successful, but on the web, nothing is show apart of two error messages. These were logged:

Undefined array key -2 at /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php#272 
{
  "reqId": "APMX0Z3SEmTLfDXsiAEl",
  "level": 3,
  "time": "2024-01-17T10:31:50+00:00",
  "remoteAddr": "127.0.0.1",
  "user": "master",
  "app": "PHP",
  "method": "GET",
  "url": "/master/index.php/apps/tables/table",
  "message": "Undefined array key -2 at /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php#272",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0",
  "version": "29.0.0.3",
  "data": {
    "app": "PHP"
  }
}
OCA\\Tables\\Db\\Row2Mapper::getFilterExpression(): Argument #2 ($column) must be of type OCA\\Tables\\Db\\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272 in file '/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php' line 282 
{
  "reqId": "APMX0Z3SEmTLfDXsiAEl",
  "level": 3,
  "time": "2024-01-17T10:31:50+00:00",
  "remoteAddr": "127.0.0.1",
  "user": "master",
  "app": "index",
  "method": "GET",
  "url": "/master/index.php/apps/tables/table",
  "message": "OCA\\Tables\\Db\\Row2Mapper::getFilterExpression(): Argument #2 ($column) must be of type OCA\\Tables\\Db\\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272 in file '/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php' line 282",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0",
  "version": "29.0.0.3",
  "exception": {
    "Exception": "Exception",
    "Message": "OCA\\Tables\\Db\\Row2Mapper::getFilterExpression(): Argument #2 ($column) must be of type OCA\\Tables\\Db\\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272 in file '/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php' line 282",
    "Code": 0,
    "Trace": [
      {
        "file": "/srv/http/nextcloud/master/lib/private/AppFramework/App.php",
        "line": 184,
        "function": "dispatch",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          [
            "OCA\\Tables\\Controller\\TableController"
          ],
          "index"
        ]
      },
      {
        "file": "/srv/http/nextcloud/master/lib/private/Route/Router.php",
        "line": 315,
        "function": "main",
        "class": "OC\\AppFramework\\App",
        "type": "::",
        "args": [
          "OCA\\Tables\\Controller\\TableController",
          "index",
          [
            "OC\\AppFramework\\DependencyInjection\\DIContainer"
          ],
          [
            "tables.table.index"
          ]
        ]
      },
      {
        "file": "/srv/http/nextcloud/master/lib/base.php",
        "line": 1069,
        "function": "match",
        "class": "OC\\Route\\Router",
        "type": "->",
        "args": [
          "/apps/tables/table"
        ]
      },
      {
        "file": "/srv/http/nextcloud/master/index.php",
        "line": 39,
        "function": "handleRequest",
        "class": "OC",
        "type": "::",
        "args": []
      }
    ],
    "File": "/srv/http/nextcloud/master/lib/private/AppFramework/Http/Dispatcher.php",
    "Line": 170,
    "Previous": {
      "Exception": "TypeError",
      "Message": "OCA\\Tables\\Db\\Row2Mapper::getFilterExpression(): Argument #2 ($column) must be of type OCA\\Tables\\Db\\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272",
      "Code": 0,
      "Trace": [
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php",
          "line": 272,
          "function": "getFilterExpression",
          "class": "OCA\\Tables\\Db\\Row2Mapper",
          "type": "->",
          "args": [
            [
              "OC\\DB\\QueryBuilder\\QueryBuilder"
            ],
            null,
            "is-equal",
            "Prime Admin XXVIII"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php",
          "line": 258,
          "function": "getFilter",
          "class": "OCA\\Tables\\Db\\Row2Mapper",
          "type": "->",
          "args": [
            [
              "OC\\DB\\QueryBuilder\\QueryBuilder"
            ],
            [
              [
                -2,
                "is-equal",
                "Prime Admin XXVIII"
              ],
              [
                40,
                "is-empty",
                ""
              ]
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php",
          "line": 236,
          "function": "getFilterGroups",
          "class": "OCA\\Tables\\Db\\Row2Mapper",
          "type": "->",
          "args": [
            [
              "OC\\DB\\QueryBuilder\\QueryBuilder"
            ],
            [
              [
                [
                  -2,
                  "is-equal",
                  "Prime Admin XXVIII"
                ],
                [
                  40,
                  "is-empty",
                  ""
                ]
              ]
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php",
          "line": 139,
          "function": "addFilterToQuery",
          "class": "OCA\\Tables\\Db\\Row2Mapper",
          "type": "->",
          "args": [
            [
              "OC\\DB\\QueryBuilder\\QueryBuilder"
            ],
            [
              [
                [
                  -2,
                  "is-equal",
                  "Prime Admin XXVIII"
                ],
                [
                  40,
                  "is-empty",
                  ""
                ]
              ]
            ],
            "master"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php",
          "line": 674,
          "function": "getWantedRowIds",
          "class": "OCA\\Tables\\Db\\Row2Mapper",
          "type": "->",
          "args": [
            "master",
            6,
            [
              [
                [
                  -2,
                  "is-equal",
                  "@my-name"
                ],
                [
                  40,
                  "is-empty",
                  ""
                ]
              ]
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Service/RowService.php",
          "line": 516,
          "function": "countRowsForView",
          "class": "OCA\\Tables\\Db\\Row2Mapper",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Db\\View",
              1
            ],
            "master",
            [
              [
                "OCA\\Tables\\Db\\Column",
                35
              ],
              [
                "OCA\\Tables\\Db\\Column",
                36
              ],
              [
                "OCA\\Tables\\Db\\Column",
                37
              ],
              [
                "OCA\\Tables\\Db\\Column",
                38
              ],
              [
                "OCA\\Tables\\Db\\Column",
                39
              ]
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Service/ViewService.php",
          "line": 367,
          "function": "getViewRowsCount",
          "class": "OCA\\Tables\\Service\\RowService",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Db\\View",
              1
            ],
            "master"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Service/ViewService.php",
          "line": 76,
          "function": "enhanceView",
          "class": "OCA\\Tables\\Service\\ViewService",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Db\\View",
              1
            ],
            "master"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Service/TableService.php",
          "line": 198,
          "function": "findAll",
          "class": "OCA\\Tables\\Service\\ViewService",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Db\\Table",
              6
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Service/TableService.php",
          "line": 124,
          "function": "enhanceTable",
          "class": "OCA\\Tables\\Service\\TableService",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Db\\Table",
              6
            ],
            "master"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Controller/TableController.php",
          "line": 39,
          "function": "findAll",
          "class": "OCA\\Tables\\Service\\TableService",
          "type": "->",
          "args": [
            "master"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Controller/Errors.php",
          "line": 16,
          "function": "OCA\\Tables\\Controller\\{closure}",
          "class": "OCA\\Tables\\Controller\\TableController",
          "type": "->",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/apps-repos/tables/lib/Controller/TableController.php",
          "line": 38,
          "function": "handleError",
          "class": "OCA\\Tables\\Controller\\TableController",
          "type": "->",
          "args": [
            [
              "Closure"
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/lib/private/AppFramework/Http/Dispatcher.php",
          "line": 232,
          "function": "index",
          "class": "OCA\\Tables\\Controller\\TableController",
          "type": "->",
          "args": []
        },
        {
          "file": "/srv/http/nextcloud/master/lib/private/AppFramework/Http/Dispatcher.php",
          "line": 138,
          "function": "executeController",
          "class": "OC\\AppFramework\\Http\\Dispatcher",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Controller\\TableController"
            ],
            "index"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/lib/private/AppFramework/App.php",
          "line": 184,
          "function": "dispatch",
          "class": "OC\\AppFramework\\Http\\Dispatcher",
          "type": "->",
          "args": [
            [
              "OCA\\Tables\\Controller\\TableController"
            ],
            "index"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/lib/private/Route/Router.php",
          "line": 315,
          "function": "main",
          "class": "OC\\AppFramework\\App",
          "type": "::",
          "args": [
            "OCA\\Tables\\Controller\\TableController",
            "index",
            [
              "OC\\AppFramework\\DependencyInjection\\DIContainer"
            ],
            [
              "tables.table.index"
            ]
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/lib/base.php",
          "line": 1069,
          "function": "match",
          "class": "OC\\Route\\Router",
          "type": "->",
          "args": [
            "/apps/tables/table"
          ]
        },
        {
          "file": "/srv/http/nextcloud/master/index.php",
          "line": 39,
          "function": "handleRequest",
          "class": "OC",
          "type": "::",
          "args": []
        }
      ],
      "File": "/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php",
      "Line": 282
    },
    "message": "OCA\\Tables\\Db\\Row2Mapper::getFilterExpression(): Argument #2 ($column) must be of type OCA\\Tables\\Db\\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272 in file '/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php' line 282",
    "exception": {},
    "CustomMessage": "OCA\\Tables\\Db\\Row2Mapper::getFilterExpression(): Argument #2 ($column) must be of type OCA\\Tables\\Db\\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272 in file '/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php' line 282"
  }
}

@juliusknorr
Copy link
Member

Notes from the migration test call as separate comments so they can be marked as resolved:

  • We should gracefully handle table row data where not matching column is present (just log and skip)
  • Make sure to catch those exceptions

Exception for the missing column for data

An unhandled exception has been thrown:
Error: Call to a member function getType() on null in /var/www/html/apps-shared/tables/lib/Db/Row2Mapper.php:506
Stack trace:
#0 /var/www/html/apps-shared/tables/lib/Db/Row2Mapper.php(418): OCA\Tables\Db\Row2Mapper->insertCell(259, 492, 'fbsdrgsdrgsesef...', '2023-05-31 08:0...', 'philipp')
#1 /var/www/html/apps-shared/tables/lib/Db/LegacyRowMapper.php(448): OCA\Tables\Db\Row2Mapper->insert(Object(OCA\Tables\Db\Row2), Array)
#2 /var/www/html/apps-shared/tables/lib/Migration/NewDbStructureRepairStep.php(97): OCA\Tables\Db\LegacyRowMapper->transferLegacyRow(Object(OCA\Tables\Db\LegacyRow), Array)
#3 /var/www/html/apps-shared/tables/lib/Migration/NewDbStructureRepairStep.php(74): OCA\Tables\Migration\NewDbStructureRepairStep->transferTable(Object(OCA\Tables\Db\Table), Object(OC\Repair))
#4 /var/www/html/apps-shared/tables/lib/Migration/NewDbStructureRepairStep.php(61): OCA\Tables\Migration\NewDbStructureRepairStep->transferDataForTables(Array, Object(OC\Repair))
#5 /var/www/html/lib/private/Repair.php(127): OCA\Tables\Migration\NewDbStructureRepairStep->run(Object(OC\Repair))
#6 /var/www/html/lib/private/legacy/OC_App.php(836): OC\Repair->run()
#7 /var/www/html/lib/private/legacy/OC_App.php(777): OC_App::executeRepairSteps('tables', Array)
#8 /var/www/html/lib/private/Updater.php(360): OC_App::updateApp('tables')
#9 /var/www/html/lib/private/Updater.php(272): OC\Updater->doAppUpgrade()
#10 /var/www/html/lib/private/Updater.php(140): OC\Updater->doUpgrade('29.0.0.3', '29.0.0.3')
#11 /var/www/html/core/Command/Upgrade.php(222): OC\Updater->upgrade()
#12 /var/www/html/3rdparty/symfony/console/Command/Command.php(298): OC\Core\Command\Upgrade->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Upgrade), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/html/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/html/lib/private/Console/Application.php(206): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /var/www/html/console.php(100): OC\Console\Application->run()
#18 /var/www/html/occ(11): require_once('/var/www/html/c...')
#19 {main}

@juliusknorr
Copy link
Member

  • Also seen the exception from blizzz: OCA\Tables\Db\Row2Mapper::getFilterExpression(): Argument Adding license to readme #2 ($column) must be of type OCA\Tables\Db\Column, null given, called in /srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php on line 272 in file '/srv/http/nextcloud/master/apps-repos/tables/lib/Db/Row2Mapper.php' line 282

@juliusknorr
Copy link
Member

  • OCA\Tables\Db\Row2Mapper - insert: columns are missing where a row has no data (row id 3866):
{
  "reqId": "r87aROqirnp43e093lvU",
  "level": 3,
  "time": "2024-01-17T13:06:29+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "tables",
  "method": "",
  "url": "--",
  "message": "OCA\\Tables\\Db\\Row2Mapper - insert: columns are missing",
  "userAgent": "--",
  "version": "29.0.0.3",
  "exception": {
    "Exception": "OCA\\Tables\\Errors\\InternalError",
    "Message": "OCA\\Tables\\Db\\Row2Mapper - insert: columns are missing",
    "Code": 0,
    "Trace": [
      {
        "file": "/var/www/html/apps-shared/tables/lib/Db/LegacyRowMapper.php",
        "line": 448,
        "function": "insert",
        "class": "OCA\\Tables\\Db\\Row2Mapper",
        "type": "->",
        "args": [
          [
            "OCA\\Tables\\Db\\Row2"
          ],
          []
        ]
      },
      {
        "file": "/var/www/html/apps-shared/tables/lib/Migration/NewDbStructureRepairStep.php",
        "line": 97,
        "function": "transferLegacyRow",
        "class": "OCA\\Tables\\Db\\LegacyRowMapper",
        "type": "->",
        "args": [
          [
            "OCA\\Tables\\Db\\LegacyRow",
            3866
          ],
          []
        ]
      },
      {
        "file": "/var/www/html/apps-shared/tables/lib/Migration/NewDbStructureRepairStep.php",
        "line": 74,
        "function": "transferTable",
        "class": "OCA\\Tables\\Migration\\NewDbStructureRepairStep",
        "type": "->",
        "args": [
          [
            "OCA\\Tables\\Db\\Table",
            200
          ],
          [
            "OC\\Repair"
          ]
        ]
      },
      {
        "file": "/var/www/html/apps-shared/tables/lib/Migration/NewDbStructureRepairStep.php",
        "line": 61,
        "function": "transferDataForTables",
        "class": "OCA\\Tables\\Migration\\NewDbStructureRepairStep",
        "type": "->",
        "args": [
          [
            [
              "OCA\\Tables\\Db\\Table",
              4
            ],
            [
              "OCA\\Tables\\Db\\Table",
              5
            ],
            [
              "OCA\\Tables\\Db\\Table",
              6
            ],
            [
              "OCA\\Tables\\Db\\Table",
              7
            ],
            [
              "OCA\\Tables\\Db\\Table",
              12
            ],
            [
              "OCA\\Tables\\Db\\Table",
              14
            ],
            [
              "OCA\\Tables\\Db\\Table",
              15
            ],
            [
              "OCA\\Tables\\Db\\Table",
              16
            ],
            [
              "OCA\\Tables\\Db\\Table",
              19
            ],
            [
              "OCA\\Tables\\Db\\Table",
              22
            ],
            [
              "OCA\\Tables\\Db\\Table",
              30
            ],
            [
              "OCA\\Tables\\Db\\Table",
              34
            ],
            [
              "OCA\\Tables\\Db\\Table",
              35
            ],
            [
              "OCA\\Tables\\Db\\Table",
              37
            ],
            [
              "OCA\\Tables\\Db\\Table",
              41
            ],
            [
              "OCA\\Tables\\Db\\Table",
              43
            ],
            [
              "OCA\\Tables\\Db\\Table",
              46
            ],
            [
              "OCA\\Tables\\Db\\Table",
              49
            ],
            [
              "OCA\\Tables\\Db\\Table",
              51
            ],
            [
              "OCA\\Tables\\Db\\Table",
              52
            ],
            [
              "OCA\\Tables\\Db\\Table",
              54
            ],
            [
              "OCA\\Tables\\Db\\Table",
              56
            ],
            [
              "OCA\\Tables\\Db\\Table",
              60
            ],
            [
              "OCA\\Tables\\Db\\Table",
              61
            ],
            [
              "OCA\\Tables\\Db\\Table",
              66
            ],
            [
              "OCA\\Tables\\Db\\Table",
              68
            ],
            [
              "OCA\\Tables\\Db\\Table",
              69
            ],
            [
              "OCA\\Tables\\Db\\Table",
              70
            ],
            [
              "OCA\\Tables\\Db\\Table",
              72
            ],
            [
              "OCA\\Tables\\Db\\Table",
              76
            ],
            [
              "OCA\\Tables\\Db\\Table",
              77
            ],
            [
              "OCA\\Tables\\Db\\Table",
              78
            ],
            [
              "OCA\\Tables\\Db\\Table",
              79
            ],
            [
              "OCA\\Tables\\Db\\Table",
              80
            ],
            [
              "OCA\\Tables\\Db\\Table",
              83
            ],
            [
              "OCA\\Tables\\Db\\Table",
              84
            ],
            [
              "OCA\\Tables\\Db\\Table",
              87
            ],
            [
              "OCA\\Tables\\Db\\Table",
              88
            ],
            [
              "OCA\\Tables\\Db\\Table",
              89
            ],
            [
              "OCA\\Tables\\Db\\Table",
              90
            ],
            [
              "OCA\\Tables\\Db\\Table",
              91
            ],
            [
              "OCA\\Tables\\Db\\Table",
              92
            ],
            [
              "OCA\\Tables\\Db\\Table",
              96
            ],
            [
              "OCA\\Tables\\Db\\Table",
              100
            ],
            [
              "OCA\\Tables\\Db\\Table",
              101
            ],
            [
              "OCA\\Tables\\Db\\Table",
              102
            ],
            [
              "OCA\\Tables\\Db\\Table",
              104
            ],
            [
              "OCA\\Tables\\Db\\Table",
              105
            ],
            [
              "OCA\\Tables\\Db\\Table",
              107
            ],
            [
              "OCA\\Tables\\Db\\Table",
              108
            ],
            [
              "OCA\\Tables\\Db\\Table",
              109
            ],
            [
              "OCA\\Tables\\Db\\Table",
              110
            ],
            [
              "OCA\\Tables\\Db\\Table",
              111
            ],
            [
              "OCA\\Tables\\Db\\Table",
              112
            ],
            [
              "OCA\\Tables\\Db\\Table",
              114
            ],
            [
              "OCA\\Tables\\Db\\Table",
              115
            ],
            [
              "OCA\\Tables\\Db\\Table",
              116
            ],
            [
              "OCA\\Tables\\Db\\Table",
              117
            ],
            [
              "OCA\\Tables\\Db\\Table",
              119
            ],
            [
              "OCA\\Tables\\Db\\Table",
              121
            ],
            [
              "OCA\\Tables\\Db\\Table",
              122
            ],
            [
              "OCA\\Tables\\Db\\Table",
              123
            ],
            [
              "OCA\\Tables\\Db\\Table",
              124
            ],
            [
              "OCA\\Tables\\Db\\Table",
              125
            ],
            [
              "OCA\\Tables\\Db\\Table",
              127
            ],
            [
              "OCA\\Tables\\Db\\Table",
              129
            ],
            [
              "OCA\\Tables\\Db\\Table",
              130
            ],
            [
              "OCA\\Tables\\Db\\Table",
              131
            ],
            [
              "OCA\\Tables\\Db\\Table",
              132
            ],
            [
              "OCA\\Tables\\Db\\Table",
              133
            ],
            [
              "OCA\\Tables\\Db\\Table",
              134
            ],
            [
              "OCA\\Tables\\Db\\Table",
              135
            ],
            [
              "OCA\\Tables\\Db\\Table",
              136
            ],
            [
              "OCA\\Tables\\Db\\Table",
              138
            ],
            [
              "OCA\\Tables\\Db\\Table",
              139
            ],
            [
              "OCA\\Tables\\Db\\Table",
              140
            ],
            [
              "OCA\\Tables\\Db\\Table",
              141
            ],
            [
              "OCA\\Tables\\Db\\Table",
              142
            ],
            [
              "OCA\\Tables\\Db\\Table",
              143
            ],
            [
              "OCA\\Tables\\Db\\Table",
              144
            ],
            [
              "OCA\\Tables\\Db\\Table",
              145
            ],
            [
              "OCA\\Tables\\Db\\Table",
              146
            ],
            [
              "OCA\\Tables\\Db\\Table",
              148
            ],
            [
              "OCA\\Tables\\Db\\Table",
              149
            ],
            [
              "OCA\\Tables\\Db\\Table",
              150
            ],
            [
              "OCA\\Tables\\Db\\Table",
              151
            ],
            [
              "OCA\\Tables\\Db\\Table",
              152
            ],
            [
              "OCA\\Tables\\Db\\Table",
              153
            ],
            [
              "OCA\\Tables\\Db\\Table",
              154
            ],
            [
              "OCA\\Tables\\Db\\Table",
              155
            ],
            [
              "OCA\\Tables\\Db\\Table",
              156
            ],
            [
              "OCA\\Tables\\Db\\Table",
              159
            ],
            [
              "OCA\\Tables\\Db\\Table",
              160
            ],
            [
              "OCA\\Tables\\Db\\Table",
              161
            ],
            [
              "OCA\\Tables\\Db\\Table",
              163
            ],
            [
              "OCA\\Tables\\Db\\Table",
              164
            ],
            [
              "OCA\\Tables\\Db\\Table",
              165
            ],
            [
              "OCA\\Tables\\Db\\Table",
              166
            ],
            [
              "OCA\\Tables\\Db\\Table",
              170
            ],
            [
              "OCA\\Tables\\Db\\Table",
              171
            ],
            [
              "OCA\\Tables\\Db\\Table",
              172
            ],
            [
              "OCA\\Tables\\Db\\Table",
              173
            ],
            [
              "OCA\\Tables\\Db\\Table",
              174
            ],
            [
              "OCA\\Tables\\Db\\Table",
              175
            ],
            [
              "OCA\\Tables\\Db\\Table",
              176
            ],
            [
              "OCA\\Tables\\Db\\Table",
              177
            ],
            [
              "OCA\\Tables\\Db\\Table",
              182
            ],
            [
              "OCA\\Tables\\Db\\Table",
              183
            ],
            [
              "OCA\\Tables\\Db\\Table",
              184
            ],
            [
              "OCA\\Tables\\Db\\Table",
              185
            ],
            [
              "OCA\\Tables\\Db\\Table",
              186
            ],
            [
              "OCA\\Tables\\Db\\Table",
              188
            ],
            [
              "OCA\\Tables\\Db\\Table",
              189
            ],
            [
              "OCA\\Tables\\Db\\Table",
              190
            ],
            [
              "OCA\\Tables\\Db\\Table",
              192
            ],
            [
              "OCA\\Tables\\Db\\Table",
              197
            ],
            [
              "OCA\\Tables\\Db\\Table",
              198
            ],
            [
              "OCA\\Tables\\Db\\Table",
              200
            ],
            [
              "OCA\\Tables\\Db\\Table",
              201
            ],
            [
              "OCA\\Tables\\Db\\Table",
              203
            ],
            [
              "OCA\\Tables\\Db\\Table",
              204
            ],
            [
              "OCA\\Tables\\Db\\Table",
              206
            ],
            [
              "OCA\\Tables\\Db\\Table",
              209
            ],
            [
              "OCA\\Tables\\Db\\Table",
              210
            ],
            [
              "OCA\\Tables\\Db\\Table",
              212
            ],
            [
              "OCA\\Tables\\Db\\Table",
              213
            ],
            [
              "OCA\\Tables\\Db\\Table",
              214
            ],
            [
              "OCA\\Tables\\Db\\Table",
              215
            ],
            [
              "OCA\\Tables\\Db\\Table",
              216
            ],
            [
              "OCA\\Tables\\Db\\Table",
              217
            ],
            [
              "OCA\\Tables\\Db\\Table",
              218
            ],
            [
              "OCA\\Tables\\Db\\Table",
              219
            ],
            [
              "OCA\\Tables\\Db\\Table",
              220
            ],
            [
              "OCA\\Tables\\Db\\Table",
              221
            ],
            [
              "OCA\\Tables\\Db\\Table",
              222
            ],
            [
              "OCA\\Tables\\Db\\Table",
              223
            ],
            [
              "OCA\\Tables\\Db\\Table",
              224
            ],
            [
              "OCA\\Tables\\Db\\Table",
              226
            ],
            [
              "OCA\\Tables\\Db\\Table",
              229
            ],
            [
              "OCA\\Tables\\Db\\Table",
              230
            ],
            [
              "OCA\\Tables\\Db\\Table",
              235
            ],
            [
              "OCA\\Tables\\Db\\Table",
              236
            ],
            [
              "OCA\\Tables\\Db\\Table",
              239
            ],
            [
              "OCA\\Tables\\Db\\Table",
              240
            ],
            [
              "OCA\\Tables\\Db\\Table",
              241
            ],
            [
              "OCA\\Tables\\Db\\Table",
              242
            ],
            [
              "OCA\\Tables\\Db\\Table",
              243
            ],
            [
              "OCA\\Tables\\Db\\Table",
              244
            ],
            [
              "OCA\\Tables\\Db\\Table",
              245
            ],
            [
              "OCA\\Tables\\Db\\Table",
              246
            ]
          ],
          [
            "OC\\Repair"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/private/Repair.php",
        "line": 127,
        "function": "run",
        "class": "OCA\\Tables\\Migration\\NewDbStructureRepairStep",
        "type": "->",
        "args": [
          [
            "OC\\Repair"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/private/legacy/OC_App.php",
        "line": 836,
        "function": "run",
        "class": "OC\\Repair",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/lib/private/legacy/OC_App.php",
        "line": 777,
        "function": "executeRepairSteps",
        "class": "OC_App",
        "type": "::",
        "args": [
          "*** sensitive parameters replaced ***",
          [
            "OCA\\Tables\\Migration\\NewDbStructureRepairStep"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/private/Updater.php",
        "line": 360,
        "function": "updateApp",
        "class": "OC_App",
        "type": "::",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/html/lib/private/Updater.php",
        "line": 272,
        "function": "doAppUpgrade",
        "class": "OC\\Updater",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/lib/private/Updater.php",
        "line": 140,
        "function": "doUpgrade",
        "class": "OC\\Updater",
        "type": "->",
        "args": [
          "29.0.0.3",
          "29.0.0.3"
        ]
      },
      {
        "file": "/var/www/html/core/Command/Upgrade.php",
        "line": 222,
        "function": "upgrade",
        "class": "OC\\Updater",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/3rdparty/symfony/console/Command/Command.php",
        "line": 298,
        "function": "execute",
        "class": "OC\\Core\\Command\\Upgrade",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Console\\Input\\ArgvInput"
          ],
          [
            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
          ]
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/console/Application.php",
        "line": 1040,
        "function": "run",
        "class": "Symfony\\Component\\Console\\Command\\Command",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Console\\Input\\ArgvInput"
          ],
          [
            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
          ]
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/console/Application.php",
        "line": 301,
        "function": "doRunCommand",
        "class": "Symfony\\Component\\Console\\Application",
        "type": "->",
        "args": [
          [
            "OC\\Core\\Command\\Upgrade"
          ],
          [
            "Symfony\\Component\\Console\\Input\\ArgvInput"
          ],
          [
            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
          ]
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/console/Application.php",
        "line": 171,
        "function": "doRun",
        "class": "Symfony\\Component\\Console\\Application",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Console\\Input\\ArgvInput"
          ],
          [
            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/private/Console/Application.php",
        "line": 206,
        "function": "run",
        "class": "Symfony\\Component\\Console\\Application",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Console\\Input\\ArgvInput"
          ],
          [
            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
          ]
        ]
      },
      {
        "file": "/var/www/html/console.php",
        "line": 100,
        "function": "run",
        "class": "OC\\Console\\Application",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/occ",
        "line": 11,
        "args": [
          "/var/www/html/console.php"
        ],
        "function": "require_once"
      }
    ],
    "File": "/var/www/html/apps-shared/tables/lib/Db/Row2Mapper.php",
    "Line": 403,
    "message": "OCA\\Tables\\Db\\Row2Mapper - insert: columns are missing",
    "exception": {},
    "CustomMessage": "OCA\\Tables\\Db\\Row2Mapper - insert: columns are missing"
  }
}

Table has no columns but tons of empty rows:

Screenshot 2024-01-17 at 15 32 39

@juliusknorr juliusknorr mentioned this pull request Jan 19, 2024
Closed
@juliusknorr

This comment was marked as resolved.

Sign in to view
Florian Steffens added 4 commits January 29, 2024 11:04
add migration file for new db tables (only for text-line and number c…
c42bd25 
…olumns yet)

Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
small adjustments
308538e 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
new models and mappers
cf70afb 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
basic integration of the new models (only for text-line and number co…
f86cb79 
…lumns yet)

Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
Florian Steffens and others added 26 commits January 29, 2024 11:04
use new row mapper in clean command
4e3e848 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
adjust the clean command to use the new db structure and code base
9c4a182 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
@blizzz
Update lib/Service/RowService.php
2951d07 
Co-authored-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
Signed-off-by: Florian <dev@d10t.de>
only add the where clause if needed
e785a9a 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
code cleanup
46be5b4 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
added some error details
2b6945e 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
Revert "added some error details"
32fb5a2 
This reverts commit b87f19c.
adjust psalm return types
b08ed0b 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
add CI upgrade test
6dd64f6 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
add unit test for data migration to new db modal
baf54ca 
- prepare LegacyRowMapper.php and Row2.php
- add LegacyRowMapperTest.php

Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
simplify the columns definition helper
e84c102 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
only trigger CI upgrade test on PR with destination branch main
152ed10 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
remove unnecessary method
1bdc6da 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
fix permission check
bb74f19 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
Add command for clean up legacy rows
1cdcbac 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
cleanup
aeae31a 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
add handling for view meta-column filtering
85f2e5b 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
@juliusknorr
Update lib/Migration/NewDbStructureRepairStep.php
bead04e 
Co-authored-by: Julius Härtl <jus@bitgrid.net>
Signed-off-by: Florian <dev@d10t.de>
Check for column existence
41f2174 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
Adjust the unit tests
ce72ade 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
fix: replace placeholder for selection and check columns
3c9d453 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>

fixup! fix: replace placeholder for selection and check columns
Adjust no-delete option to delete option to avoid data lost
89ca33d 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
@juliusknorr
fix: Print error during commands and catch more general errors during…
57e42b7 
… migration

Signed-off-by: Julius Härtl <jus@bitgrid.net>
@juliusknorr
fix: use all rows for querying row candidates if a filter on a not-ac…
72b9ef3 
…cessible row is used

Signed-off-by: Julius Härtl <jus@bitgrid.net>
enh: add filtering in views for multi selection
02a58d0 
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
test: add end2end tests for views filtering for selection columns
55e9c29 
- add data-cy to form sections
- add general commands to fill in data in forms for selection column
- add test cases for all selection types
- add basic combination test cases
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>

fix: test urls
Signed-off-by: Florian Steffens <florian.steffens@nextcloud.com>
@datenangebot datenangebot merged commit 8818266 into main Jan 29, 2024
46 checks passed
@datenangebot datenangebot deleted the refactor/db branch January 29, 2024 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@blizzz blizzz blizzz left review comments

@juliusknorr juliusknorr juliusknorr approved these changes

Assignees
No one assigned
Labels
2. developing Work in progress enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Change database structure to store the content
3 participants
@datenangebot @blizzz @juliusknorr